Understanding Transcriptions and Translations Generation via VIDIZMO Speech & Text Analyzer

The VIDIZMO Speech & Text Analyzer enables users to generate transcription and translation related to audio for both audio and video files. You can either specify languages or allow the application to auto-detect the language for various processes like transcription or translation. This article will explain how these AI insights are generated and how the application performs its functionality.

Concept

When you choose an audio or video file for processing with Auto-Detect selected, the AI model set up in the VIDIZMO Speech & Text Analyzer app detects and predicts the language during playback. Once the language is detected and predicted, the app will generate the transcript in that language for the entire playback duration of the audio or video file.

The accuracy of transcriptions or translation for a language depends on that specific language's word error rate (WER). A lower WER indicates that the transcriptions generated for that language are likely to be accurate. In addition, your application can also impact the accuracy of the transcriptions generated.

Here is a list of languages supported by the VIDIZMO Speech & Text Analyzer, along with their respective WER when the Large-V1 model is used. The performance of the Speech & Text Analyzer and the accuracy of its Insights vary depending on the model size. The following sections of this article provide more information about the models. The languages in this list arranged from most accurate (least WER) to less accurate (most WER).

Serial No.	Language	WER
1	Spanish	3.5
2	Italian	4.2
3	English	4.5
4	Portuguese	4.8
5	German	5.5
6	Japanese	6.4
7	Russian	6.4
8	Polish	7.2
9	French	7.7
10	Catalan	8.0
11	Dutch	8.3
12	Indonesian	8.5
13	Turkmen	9.4
14	Turkish	9.4
15	Malay	10.2
16	Ukrainian	10.3
17	Swedish	10.5
18	Vietnamese	10.7
19	Norwegian	11.4
20	Finnish	12.2
21	Thai	13.2
22	Korean	15.2
23	Romanian	15.4
24	Slovak	15.7
25	Tagalog	15.8
26	Croatian	16.7
27	Danish	16.8
28	Czech	17.4
29	Arabic	18.1
30	Bulgarian	18.4
31	Greek	18.7
32	Galician	19.0
33	Chinese	19.6
34	Macedonian	20.6
35	Tamil	20.6
36	Bosnian	20.7
37	Hungarian	21.0
38	Urdu	25.0
39	Estonian	25.5
40	Hindi	26.9
41	Slovenian	27.8
42	Latvian	28.3
43	Azerbaijani	28.7
44	Serbian	29.2
45	Hebrew	30.2
46	Lithuanian	35.2
47	Persian	36.1
48	Welsh	36.6
49	Afrikaans	42.6
50	Icelandic	43.0
51	Marathi	43.7
52	Kazakh	43.8
53	Māori	45.7
54	Swahili	47.9
55	Nepali	52.2
56	Armenian	53.7
57	Belarusian	56.6
58	Kannada	69.8
59	Tajik	74.5
60	Occitan	75.9
61	Lingala	76.8
62	Maltese	80.5
63	Luxembourgish	86.5
64	Hausa	87.0
65	Javanese	87.0
66	Pashto	92.7
67	Uzbek	93.3
68	Khmer	96.0
69	Georgian	100.5
70	Telugu	100.6
71	Malayalam	101.4
72	Lao	101.6
73	Punjabi	102.8
74	Somali	103.5
75	Gujarati	103.9
76	Bengali	104.9
77	Assamese	105.6
78	Mongolian	106.2
79	Yoruba	111.7
80	Amharic	129.3
81	Shona	130.0
82	Sindhi	177.9

In addition to the languages above, the VIDIZMO Speech & Text Analyzer has the capacity to generate insights for some additional rare languages. However, due to their scarcity and inadequate training data, the estimated Word Error Rate (WER) for these languages is high and may create unusual or insufficient results.

-Bashkir -Tibetan -Breton -Basque -Faroese -Hawaiian -Haitian Creole -Latin -Malagasy -Burmese -Norwegian Nynorsk -Sanskrit -Sinhalese -Albanian -Sundanese -Tatar -Yiddish -Cantonese

The Speech & Text Analyzer application exclusively handles videos that have audio. If speech is detected in that audio, the app generates transcripts accordingly. When the input is a video, the VIDIZMO Speech & Text Analyzer separates the audio component from the video and then performs the transcription process solely on the audio portion.

When the VIDIZMO Speech & Text Analyzer has Transcriptions in its Insights, you can’t use the AWS Indexer App or Azure Video Analyzer ARM for transcriptions. Additionally, you can't activate the Speech & Text Analyzer app, if either the AWS Indexer App or Azure Video Analyzer ARM are enabled with the option to generate Transcriptions.

Translation Concept

If the user has also opted for Translation, the VIDIZMO Speech & Text Analyzer app will translate the detected language in the audio or video file. These translations will be present in the transcription pane of your file's playback page. If both Transcription and Translations are selected, the Speech & Text Analyzer generates both Insights which the user can choose to see from the Transcription pane. As of now, the VIDIZMO Speech & Text Analyzer can only perform Translations in English.

If an audio or video file doesn't have any transcriptions, processing it with the Translation Insight generates the English translations, regardless of the spoken language in it. Portal content that already has transcriptions, either from another indexing application or the user uploading a .vtt file, can still be processed for the English translations.

The features offered by the VIDIZMO Speech & Text Analyzer app utilize AI processing as a consumption metric for your VIDIZMO Account. To learn how you can view consumption reports, refer to Consumption Reports for SaaS Deployment Overview.

Processing

To generate accurate transcriptions or translations for your audio and video files, the Speech & Text Analyzer aims to minimize substitutions, insertions, and deletions, which contributes to reducing the overall Word Error Rate (WER). The less the WER of a transcription, the more accurate it will be.

In the first step of the indexing process, the raw audio inputs from your audio or video files are converted into a log-Mel spectrogram using a feature extractor. The system then maps these audio spectrogram features to a sequence of text tokens that have encoder-hidden states. These text tokens are then decoded regressively by an internal language model (LM).

You can also configure the application to automatically process any audio or video file uploaded or added to your Portal via other means. Learn how to do this configuration and more by visiting: Configuring VIDIZMO Speech & Text Analyzer for Transcription and Translation

Conclusion

The VIDIZMO Speech & Text Analyzer aims to generate reliable, accurate transcriptions, translations and AI insights for all your audio and video files. You can also configure the application to automatically generate insights for files that you add to the Portal using the same configured settings. You can also manually create and regenerate transcriptions with different parameters from your Portal if you are unsatisfied with the results.

For a guide to a more practical or hands-on approach to the transcription generation process, visit How to Generate Transcriptions and Translations using VIDIZMO Speech & Text Analyzer